Methods and systems for treating autism

ABSTRACT

A method for treating autism is provided. The method includes presenting to affected subjects therapeutic content in the form of images or video in a virtual or augmented reality system and monitoring in real time the behaviors and responses of the subject to the therapy. The virtual or augmented reality system may further include audio, and the monitoring of the therapy may be achieved using one or more tracking sensors, such as a camera.

CROSS-REFERENCE

This application is a continuation of U.S. patent application Ser. No.15/782,216, filed Oct. 12, 2017, which claims the benefit of U.S.Provisional Application No. 62/408,663, filed Oct. 14, 2016, each ofwhich applications are entirely incorporated herein by reference.

BACKGROUND

Autism, also known as autism spectrum disorder (ASD), refers to acomplex neurodevelopmental disorder in which affected subjects commonlyshow difficulties engaging in social interactions and communication andrepetitive behaviors. Autism may affect millions of people in the UnitedStates alone, and is becoming more prevalent in the population everyyear. Although there is no known cure, today, autism therapy rangesacross a myriad of specialties including neurology, psychiatry, physicaltherapy, occupational therapy, behavioral therapy, and speech languagepathology. Such therapeutic methods can help autistic individuals gainsocial and communication skills.

Virtual or augmented reality is the coordinated experience of a user inan environment manufactured by and housed in computer processors andmemory, where the environment is coordinated with the user's motion anduse of the senses such as sight, sound, and touch. Virtual realitysystems are often used to create simulations so that a user may learn ortrain while being exposed to little or no physical risk.

SUMMARY

Recognized herein is the need for improved methods and systems fortreating a mental or neurodevelopmental disorder, such as autism. Suchmethods and systems can employ the use of virtual or augmented realitysystems.

The present disclosure provides methods and systems for treating orsupplementing the treatment of a subject for a mental orneurodevelopmental disorder, such as autism, using a virtual oraugmented reality system. The method can comprise placing the subject ina virtual world using a virtual or augmented reality system thatprovides visual, auditory, and/or haptic stimulation, monitoring thesubject's interaction with the virtual and real world, and measuring thesubject's progress toward one or more therapeutic goals. The virtual oraugmented reality system can comprise a display screen for visualengagement, headphones or speakers for auditory stimulation, andcontrollers for physical input as well as haptic feedback. Monitoringmay be in real-time

The method may further comprise the subject interacting with the virtualworld using eye movement, head movement, and/or one or more controllersattached to a body and/or limb of the subject. One or more sensorsintegrated or external to the virtual or augmented reality system maydetect the subject's interactions with the virtual world.

The method may further comprise the subject interacting with atherapist, parent, or peer to progress toward the one or moretherapeutic goals. For example, the subject may be paired with thetherapist, parent, or peer, or a plurality of therapists, parents, orpeers, or any combination of the above. In some instances, a pairedindividual or entity may be capable of influencing the subject'sexperience in the virtual world, such as by creating, modifying, and/orremoving the visual, auditory, and/or haptic stimulations provided tothe subject in the virtual world.

In some instances, the virtual or augmented reality experiences andtherapies can be tailored to the needs of individual subjects eithermanually by a human expert (e.g., therapist) or using computeralgorithms or artificial intelligence. For example, the tailoring can beperformed based at least on prior conditions of the subject and/or basedon data collected throughout the subject's and others' use of thesystem.

In some instances, different virtual or augmented reality experience maybe organized in a library and be available for prescription orrecommendation by the human expert or computer algorithms (e.g.,artificial agent) to care givers, parents, or the subjects themselves.

In an aspect, provided is a method for treating or supplementingtreatment of a subject with mental or developmental conditions,comprising: (a) placing the subject in a virtual world using a virtualor augmented reality device that provides visual, auditory, or hapticstimulation; (b) presenting the subject's virtual or augmented realityexperience on a display external to the virtual or augmented realitydevice; (c) generating, via one or more sensors operatively coupled tothe virtual or augmented reality device, sensory data of the subject inresponse to the visual, auditory, or haptic stimulation, wherein thesensory data is indicative of the subject's progress toward one or moretherapeutic goals; and (d) evaluating at least the subject's virtual oraugmented reality experience presented on the display and the sensorydata to determine the subject's progress toward one or more therapeuticgoals.

In some embodiments, the subject is capable of interacting with thevirtual world using eye movement, head movement, or one or morecontrollers attached to a body and/or limb of the subject.

In some embodiments, the subject is capable of interacting with atherapist, parent, or peer to progress toward the one or moretherapeutic goals.

In some embodiments, the method can further comprise prescribing the oneor more therapeutic goals to the subject.

In some embodiments, the therapist, parent, or peer is located remotefrom the subject.

In some embodiments, the visual, auditory, or haptic stimulation isprovided by the therapist, parent or peer.

In some embodiments, the method is performed without direct supervisionof the subject.

In another aspect, provided is a method for treating a subject forautism, wherein the subject is or is suspected of being autistic,comprising: (a) presenting one or more images or video to the subjectusing a virtual or augmented reality system, wherein the virtual oraugmented reality system comprises a first display screen for presentingthe one or more images or video to the subject; (b) generating, via oneor more sensors operatively coupled to the virtual or augmented realitydevice, sensory data of the subject in response to the one or moreimages or video, wherein the sensory data is indicative of the subject'sprogress toward one or more therapeutic goals; and (c) evaluating atleast the sensory data to determine the subject's progress toward one ormore therapeutic goals.

In some embodiments, the subject is further presented with audio usingthe virtual or augmented reality system.

In some embodiments, the one or more sensors comprise at least one of agroup consisting of cameras, microphones, motion sensors, heat sensors,inertial sensors, and touch sensors.

In some embodiments, the subject is capable of interacting with the oneor more images or video using eye movement, head movement, or one ormore controllers attached to a body and/or limb of the subject.

In some embodiments, the subject is capable of interacting with atherapist, parent, or peer to progress toward the one or moretherapeutic goals.

In some embodiments, the therapist, parent, or peer is located remotefrom the subject.

In some embodiments, the one or more images or video is presented by thetherapist, parent or peer.

In some embodiments, the evaluating comprises comparing the sensory datato expected data for the one or more therapeutic goals.

In another aspect, provided is a method for diagnosing a user withmental or developmental conditions, comprising: (a) placing a pluralityof subjects having a first mental or developmental condition in avirtual world using one or more virtual or augmented reality devices,wherein the virtual world provides one or more of visual, auditory, andhaptic stimulation; (b) generating, via a first set of one or moresensors operatively coupled to the one or more virtual or augmentedreality devices, a first set of sensory data of the plurality ofsubjects in response to the one or more of visual, auditory, and hapticstimulation, wherein the first set of sensory data is indicative of thefirst mental or developmental condition; (c) placing the user in thevirtual world using a virtual or augmented reality device; (d)generating, via a second set of one or more sensors operatively coupledto the virtual or augmented reality device, a second set of sensory dataof the user in response to the one or more of visual, auditory, andhaptic stimulation; and (e) comparing the second set of sensory data tothe first set of sensory data to diagnose the user with the first mentalor developmental condition.

In some embodiments, the plurality of subjects is capable of interactingwith a therapist, parent, or peer in the virtual world.

In some embodiments, the therapist, parent, or peer is located remotefrom the plurality of subjects.

In some embodiments, the one or more of visual, auditory, and hapticstimulation is provided by the therapist, parent or peer.

In some embodiments, the method is performed without direct supervisionof the user.

Additional aspects and advantages of the present disclosure will becomereadily apparent to those skilled in this art from the followingdetailed description, wherein only illustrative embodiments of thepresent disclosure are shown and described. As will be realized, thepresent disclosure is capable of other and different embodiments, andits several details are capable of modifications in various obviousrespects, all without departing from the disclosure. Accordingly, thedrawings and description are to be regarded as illustrative in nature,and not as restrictive.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.To the extent publications and patents or patent applicationsincorporated by reference contradict the disclosure contained in thespecification, the specification is intended to supersede and/or takeprecedence over any such contradictory material.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings (also “Figure” and “FIG.” herein), of which:

FIG. 1 shows a schematic illustration of a virtual or augmented realitysystem.

FIG. 2 shows an illustration of a mobile companion application display.

FIG. 3 shows a logical model of a learning module.

FIG. 4 shows a logical model of a learning card.

FIG. 5 shows an illustration of a teaching module display for asupervisor.

FIGS. 6A-6C illustrate a sensory module content comprising a train room.FIG. 6A shows a user using a user device. FIG. 6B shows the user usingthe user device immersed in a virtual reality or augmented realityexperience. FIG. 6C shows an exemplary display of the sensory module asseen by the user.

FIG. 6D illustrates a sensory module content comprising an aquarium.

FIG. 6E illustrates a sensory module content comprising a butterflypark.

FIG. 6F illustrates a sensory module content comprising a lava room.

FIG. 6G illustrates a sensory module content comprising a coloring book.

FIG. 7 shows an illustration of a progress report generated from auser's response to an augmented reality or virtual reality experience.

FIG. 8 shows a computer control system that is programmed or otherwiseconfigured to implement methods provided herein.

DETAILED DESCRIPTION

While various embodiments of the invention have been shown and describedherein, it will be obvious to those skilled in the art that suchembodiments are provided by way of example only. Numerous variations,changes, and substitutions may occur to those skilled in the art withoutdeparting from the invention. It should be understood that variousalternatives to the embodiments of the invention described herein may beemployed.

Virtual Reality (VR) or Augmented Reality (AR) Therapy

In an aspect, provided are systems and methods for treating a subjectfor a mental or developmental disorder, such as, for example, autism,attention deficit hyperactivity disorder (ADHD), and social anxietydisorder. This may comprise using a virtual or augmented reality systemto place the subject in a virtual world, where the subject can bepresented with visual, auditory, and/or haptic stimulation, monitoringthe subject's interaction with the virtual and/or real world, andmeasuring the subject's progress toward one or more therapeutic goals.The subject's interactions, reactions, and/or responses to thestimulations presented in the virtual world can be quantified based atleast in part on sensory data measured for the subject, such as areaction time, gaze accuracy, gaze stability, response volume, and/orother forms or units of outputs by the subject. In some cases, thesubject's progress toward one or more therapeutic goals can bequantified based at least in part on such quantifications. The subject'sprogress toward one or more therapeutic goals can also be measured basedon qualitative observations made by another user monitoring thesubject's interactions, reactions, and responses in the virtual world,such as a therapist, operator, or supervisor. For example, the subject'sprogress toward one or more therapeutic goals can be measured based onsubjective and/or objective feedback from the monitoring user.

The systems and methods can be used for treating the subject for a givenmental or developmental disorder, such as autism. The systems andmethods can be used for treating a subject suffering a deficiency in agiven social skill (e.g., ability to retain attention). The virtualworld, and the one or more stimulations (e.g., visual, auditory, haptic)presented in the virtual world, can be designed to treat the givenmental developmental disorder and/or improve the given social skill. Forexample, the virtual world can be tailored for a given mental ordevelopmental disorder. Alternatively or in addition, the virtual worldcan be tailored for a given social skill. Alternatively or in addition,the virtual world can be tailored for a subject. One or more parametersand/or stimulations in a virtual world can be provided or adjusted inreal-time, such as by another user (e.g., therapist, operator,supervisor) monitoring the subject in the virtual world, for example,based on the subject's interactions, reactions, or responses (or lackthereof) to a previous stimulation.

Visual stimulation may be provided by presenting one or more images orvideo on a display screen. Auditory stimulation may be presented throughone or more headphones or speakers of the virtual or augmented realitysystem. The one or more headphones or speakers may be synchronized withimages or video provided by the display screen to the subject. Hapticstimulation may be presented through one or more controllers attached(or otherwise coupled) to the body and/or limb of the subject. Themonitoring may permit another individual, such as a supervisor, tomonitor treatment. The supervisor may be a therapist, an operator, ateacher, or a parent. The monitoring may or may not be remote. Themonitoring may be over one or more networks.

The subject may interact with the virtual world using eye movement, headmovement, or one or more controllers attached to the body and/or thelimb of the subject. To track the subject's movement, the system mayemploy one or more sensors, including one or more cameras. The systemmay employ other optical sensors, auditory sensors (e.g., microphones),touchpads, touchscreens, motion sensors, heat sensors, inertial sensors,touch sensors, or other sensors. The one or more sensors may be capableof measuring sensory data indicative of an output by a subject (e.g.,eye movement, head movement, facial expressions, speech, etc.).

The subject may interact with a therapist, supervisor (e.g., parent), orpeer to progress toward the one or more therapeutic goals. Thetherapist, supervisor, or peer can be located remotely from the subject.

The systems and methods may allow a therapist to prescribe one or moretherapy sessions to achieve one or more therapeutic goals to a subject.The subject may perform the prescribed therapy sessions, without directsupervision from the therapist prescribing the therapy sessions. In someinstances, the therapy sessions are capable of being reviewed at a latertime and place. Alternatively or in addition, the therapy sessions arecapable of being monitored directly or indirectly (e.g., remotely) inreal time. In some instances, the therapy sessions can be monitored byanother individual, such as a parent or guardian in real time or at alater time and place.

A method for treating a mental or developmental disorder may compriseusing a virtual or augmented reality system to present one or moreimages or video to the subject. The virtual or augmented reality systemcan comprise a display screen for presenting the one or more images orvideo to the subject. The method may comprise monitoring, in real time,a response of the subject to the one or more images or video presentedto the subject in the virtual or augmented reality system. As usedherein, real time can include a response time of less than 1 second,tenths of a second, hundredths of a second, or a millisecond. Real timecan include a process or operation (e.g., monitoring of an action by asubject) occurring simultaneously or substantially simultaneously withanother processor or operation (e.g., performing the action by thesubject).

The method may further comprise presenting audio to the subject in thevirtual or augmented reality system. The audio may be presented throughone or more headphones or speakers of the virtual or augmented realitysystem. The one or more headphones or speakers may be synchronized withimages or video provided by the display screen to the subject. Themethod may comprise monitoring, in real time, a response of the subjectto the audio presented to the subject in the virtual or augmentedreality system.

The method may further comprise presenting haptic output to the subjectin the virtual or augmented reality system. The haptic output may bepresented through one or more controllers of the virtual or augmentedreality system. The one or more controller may be synchronized withimages or video provided by the display screen to the subject and/or theaudio provided by the one or more headphones or speakers to the subject.

The method may further comprise monitoring the subject with the use ofone or more sensors, including one or more cameras. A camera may be acharge coupled device (CCD) camera. The camera may record a still imageor video. The image or video may be a two-dimensional image or video ora three-dimensional image or video. The system may employ other opticalsensors, auditory sensors (e.g., microphones), touchpads, touchscreens,motion sensors, heat sensors, inertial sensors, touch sensors, or othersensors. The one or more sensors may be capable of measuring sensorydata indicative of a subject's progress toward one or more therapeuticgoals, such as a reaction time, gaze stability, gaze accuracy, responsevolume, the presence or lack of a movement, the presence or lack ofspeaking, or other outputs by the subject.

The virtual or augmented reality system may be a virtual reality systemin which the subject is presented with content in an environment thatmay be separate from the surrounding of the subject. Alternatively, thevirtual or augmented reality system may be an augmented reality systemin which the subject is presented with content that may be overlaid orat least partially integrated with the environment of the subject.

Such system can comprise a display for presenting the subject withcontent. Such display can be provided to the subject through a userdevice (e.g., mobile device, computer, tablet, laptop, etc.), forinstance. The user device may or may not be portable. Such system canfurther comprise one or more headphones, earphones, or speakers forpresenting the subject with audio. The one or more headphones,earphones, or speakers may be synchronized with images or video providedby the display screen to the subject. The subject may access the virtualor augmented reality system with the use of a supplemental headgear(e.g., Google® Daydream/Cardboard, Oculus® Gear/Rift, and HTC® Vive).Such system can further comprise one or more controllers for presentingthe subject with haptic output. The controllers can be configured to,for example, vibrate. The controllers can comprise an actuator. Thecontrollers may be attached (or otherwise coupled) to one or more bodyparts of the subject. The display can receive the one or more images orvideo, the headphone, earphone, or speaker can receive the audio, andthe controllers can receive the haptic output to present to the subjectthrough a computer control system.

Using the virtual or augmented reality systems may advantageouslyprovide various therapeutic values to a subject (or user) receivingtreatment. The subject having or suspected of having aneurodevelopmental disorder, such as autism, may be treated by receivingVR or AR content through the system.

Subjects who can benefit from these methods can be divided intocharacteristic groups. One group may comprise individuals with cognitiveor perceptual conditions that need a more gentle and controlledintroduction to certain aspects of the real world. These cognitive orperceptual conditions may include mental conditions such as a variety ofphobias (e.g., arachnophobia, agoraphobia, acrophobia, social phobia andanxiety disorder), disorders involving distortions in the perception ofone's own body (e.g., anorexia nervosa), and any mental or neurologicalcondition that can cause sensory overload (e.g., ADHD). One group maycomprise individuals who have conditions that prevent them fromacquiring, or make it difficult for them to acquire, skills (e.g.,social skills) through the normal course of development or the naturalcourse of recovery from a trauma or disease. This may includeindividuals with autism, individuals with social (pragmatic)communication disorder, as well as those individuals that requirelanguage therapy, occupational therapy, behavioral therapy or physicaltherapy. One group may comprise a diverse group of individuals that canbenefit from novel intervention methods that were not possible outsideof virtual reality (VR) or augmented reality (AR). This group mayinclude individuals diagnosed with conditions that can improve by havingthe individual observe and involve oneself in conversation in a virtualworld to improve self-compassion or self-esteem (the lack of which areboth symptoms of clinical depression). The group may also include anyand all individuals in the general population that may benefit frompracticing social skills or routines in a controlled environment.

Furthermore, beneficially, the VR or AR system may facilitate datacollection during or subsequent to treatment. For example, a subject'sprogress toward one or more therapeutic goals may be measured withsignificantly higher accuracy and precision than traditional methods oftherapy. The VR or AR system may be capable of collecting data oncertain aspects of the treatment that was previously unavailable formeasurements via traditional methods. For example, the VR or AR systemmay comprise one or more integrated sensors that are capable ofmeasuring sensory data indicative of an output by a subject (e.g., eyemovement, head movement, facial expressions, speech, etc.). The one ormore sensors may be capable of measuring sensory data indicative of asubject's progress toward one or more therapeutic goals, such as areaction time, gaze stability, gaze accuracy, response volume, thepresence or lack of a movement, the presence or lack of speaking, orother outputs by the subject. For example, a user device presenting thesubject with the VR or AR environment may have integrated in the userdevice optical sensors, auditory sensors (e.g., microphones), touchpads,touchscreens, motion sensors, heat sensors, inertial sensors, touchsensors, or other sensors capable of measuring an output by the subject.A subject's progress toward achieving a therapeutic goal (e.g.,increasing ability to concentrate) may be quantified more accurately andprecisely than traditional methods (e.g., that rely on manualobservations of a subject or of a video recording of a subject). Forexample, the VR or AR system may track a patient's gaze with one or moreintegrated sensors that are much more accurate than manual observationsthat predict a direction of a gaze. Additionally, the integrated sensorsin the VR or AR system allow the measuring of a subject's interactions,reactions, and responses to a situation without interrupting, or beingexposed to, the subject who is wholly immersed in a VR or ARenvironment—for example, traditionally, the subject may become aware ofthe artificiality of a therapy session, such as via presence of anotherhuman or a camera, and change his or her behavior based on suchawareness, resulting in biased or otherwise inaccurate results.

In some cases, data collected (or recorded) for one or more subjects maybe aggregated to build behavior models for one or more conditions (e.g.,mental or developmental disorders). Such behavior models can beleveraged as diagnostic tools for users to be evaluated through the VRor AR system. For example, a plurality of behavior models for differentmental or developmental disorders can be stored in a library of behaviormodels, such as in one or more databases. A behavior model for a firsttype of developmental disorder can comprise data exhibited by one ormore subjects known to suffer from the first type of developmentaldisorder when placed in a first VR or AR environment. When a user to bediagnosed is placed in the same first VR or AR environment, or anenvironment similar to the first VR or AR environment, the datacollected on the user may be compared to the behavior model for thefirst type of developmental disorder to determine whether the user hasthe first type of developmental disorder or not, or a degree to whichthe user suffers from the first type of developmental disorder. In someinstances, the data collected for the user to be diagnosed may becompared to a plurality of behavior models to determine which one ormore conditions the user may suffer from (and to what degree). By way ofexample, the higher the % similarity between the collected data for theuser and the data stored for the behavior model, the more likely it is(and with higher degree) that the user suffers from the condition of thebehavior model.

The VR or AR system described herein may further provide a low cost,accessible therapeutic solution for subjects who have a mental ordevelopmental disorder. Alternatively or in addition, the VR or ARsystem described herein may provide educational and/or entertainmentvalue to subjects who have a mental or developmental disorder, andsubjects who do not.

VR or AR content can be keyed to virtual behavioral interventionprinciples which may be adapted from multiple approaches used by othertraditional methods of autism therapy. In some cases, naturalisticbehavioral intervention techniques may be employed. Naturalistictherapy, or natural environment teaching (“NET”), may occur in thesubject's natural environment. Methods and systems herein can integratesome key attributes of NET interventions into VR or AR therapy. In someexamples, the system may provide the user with the opportunity to selecta type of virtual reality or augmented reality content or a treatmentprogram (e.g., selecting the type of virtual environment such as the‘zoo’ or a ‘train station’) and encourage supervisors or other thirdparties monitoring the user (“supervisors”) to follow establishedtraining manuals, such as the Hanen OWL guidelines which teach theObserve, Wait, and Listen technique. Under these guidelines, supervisorsmay let the user explore the virtual environment while observingcarefully what interests the user, wait for the user to make spontaneouscomments or actions, listen carefully to what the user says or signs,and plan the next steps of therapy based on these comments or signs. Insome examples, the system may permit a user to create a communication orplay opportunity, if needed. That is, if the user does not communicateor do anything, the system can use playful obstructions or a similartechnique to create a communication or play opportunity. For instance, auser that stares at only one animal silently may begin to speak if thetherapist avatar blocks the user's view of the animal, or,alternatively, the user may start a game of trying to get past thetherapist. Once the user starts doing something, the therapy can returnto the OWL guidelines. In some examples, the system can provide the userwith prompts of varying supportiveness to complete a task if there is aspecific task to complete. In some examples, the system can reward theuser for succeeding in such tasks. Such reward may be, for example, animage, audio or video that is pleasing to the user, a pleasing stimulus,or a compliment from a third party (e.g., supervisor).

In doing so, the system may provide prompts of varying supportivenessthroughout the process. For example, the prompts may include, in theorder of least supportive to most supportive: lightweight prompts by thesoftware (e.g., a clickable item that blinks or is slightly translucentor a virtual hand suggesting where a user can move certain blocks);leading comment by a supervisor or therapist (e.g., using suggestivephrases such as “I see something interesting” or “That animal lookshungry”); verbal instruction (e.g., telling the user directly what todo); imitative model (e.g., a therapist or supervisor in real life, oralternatively as an avatar in the virtual environment, demonstrating adesired action so that the user can imitate); and physical prompts(e.g., directing the user by using hand-over-hand support through adesired action, or moving the user's head in an intended direction).Incorporating such virtual NET-style intervention, the system can createand provide VR or AR environments with varying levels of difficulty, inwhich the user always succeeds in a given task, with or withoutsupportive prompts. Through adaptations of such key principles, thesystem may provide goal-oriented therapies, such as, for example,therapies which build social connections, therapies which teach stories,and sensory-based therapies.

In some aspects, the system may provide therapies which may build socialconnections between the user and other individuals. The otherindividuals may include one or more subjects that are also undertreatment. These therapies can include the training of developmentalskills such as joint attention (e.g., eye contact), social reciprocity(e.g., taking turns), and pragmatic intentions (e.g., problem solving orcommunicating acceptance). For example, a therapy module for developinginferencing may help prepare the user to become capable of learning howto recognize and interpret facial expressions, and a therapy module fordeveloping joint attention may help prepare the user to become capableof understanding other people's perspectives. In some aspects, thesystem may provide therapies which teach functional skills usingstories. These therapies can include building a practice environment forboth routine tasks such as crossing a street and non-routine tasks suchas responding to fire alarms or participating in a Halloweentrick-or-treating activity. Other practice environments includeinteracting with a police officer that a user meets in an urban street.Beneficially, such stories provide an effective and entertaining (‘fun’)solution to teach subjects how to navigate everyday interactions byplacing the subjects in a controlled virtual or augmented realityenvironment, thereby shielding the subjects from potential actual harmsand retaining control to effectively guide the subjects at a flexiblepace.

In some aspects, the system may provide sensory-based therapies.Sensory-based therapies can include using a VR or AR environment tobuild specific calming or stimulating experiences with a focus onhelping users with neurodevelopmental disorders or disabilities, such asautism, to manage or respond to otherwise challenging environments andsituations. For example, a calming module may allow a user to play asimple and delightful musical instrument using just the direction of hisor her gaze.

Monitoring a User's Virtual Reality Experience

Through virtual reality or augmented reality, the system can providetherapy through a network or graph of interactions that are triggeredeither by one or more actions of the user or by a supervisor or otherthird party that is monitoring the user. In some instances, aninteraction, reaction, or response may be triggered by a peer (e.g.,another subject user). Every discrete action from the user can trigger acombination of animation, audio, and/or visual instructions to thesupervisor. A supervisor, who may or may not be a therapist, candirectly monitor and supervise the user receiving the VR or AR therapythrough the use of a companion application. Such monitoring may occurremotely, wherein the subject user and the therapist are physically andgeographically at a significant distance from each other (e.g.,different buildings, different countries, etc.). Alternatively or inaddition, such monitoring may occur as in-room monitoring.

The system can comprise at least one or at least two devices: one devicecapable of providing a virtual reality or augmented reality environment(e.g., smartphone, dedicated headset device like an HTC® Vive, Google®Cardboard headset) to the user, and the same device, or a second device,capable of providing a companion application to the supervisor. The atleast two devices can be communicatively coupled together, such as viatangible connection cable or wireless or Bluetooth pairing. Through thecompanion application, a supervisor may access, and, if needed,intervene in, the user's virtual or augmented reality experience. As analternative, the system can comprise a single integrated device.

FIG. 1 shows a schematic illustration of a virtual or augmented realitysystem. A user 102 may be paired with a supervisor 104. The user 102 maybe a subject. The user 102 may have a condition, such as a mental ordevelopmental disorder (e.g., autism). The user 102 may access a virtualreality or augmented reality experience 110 via a first user device. Thefirst user device may be any device comprising one or more displays,such as a monitor, screen, mobile phone, computer, smartphone, laptop,tablet, television, smart television, or other device. For example, theuser 102 may access the experience 110 through the use of supplementalheadsets 115 and 120 (e.g., Google® Daydream/Cardboard, Oculus®Gear/Rift, HTC® Vive, etc.).

The first user device may be communicatively coupled to one or moresensors, such as described elsewhere herein. The one or more sensors maybe integrated in the first user device or external to, and operativelycoupled to, the first user device, such as via wired or wireless (e.g.,Bluetooth, Wi-Fi, Near Field Communication (NFC), etc.) connections. Theone or more sensors may be capable of collecting data on the user 102,such as the user's interactions, reactions, and/or responses to one ormore components and/or stimulations in the VR or AR experience 110.

The VR or AR experience 110 may comprise one or more VR or AR scenes125. For example, the VR or AR experience 110 may comprise atime-dependent progression of one or more VR or AR scenes 125. The VR orAR scenes 125 may be dynamic, such as comprising one or more dynamiccomponents (e.g., animation, audio, etc.) and/or components that can betriggered to change. The user 102 may be capable of interacting with, orreacting to or responding to, one or more components of the VR or ARscenes 125. The user 102 may have a stereoscopical view of the one ormore VR or AR scenes 125 in the VR or AR experience 110. The VR or ARexperience 110 can be a 360° experience. The VR or AR experience 110 maybe capable of presenting one or more stimulations, such as visualstimulations, audio stimulations, and/or haptic stimulations, such asvia the first user device. Alternatively or in addition, the one or morestimulations may be provided via one or more external devicesoperatively coupled to the first user device, such as via wired orwireless connections. Such other devices can include, for example, otherdisplays, screens, speakers, headphones, earphones, controllers,actuators, lamps, or other devices capable of providing visual, audio,and/or haptic output to the user 102.

The supervisor 104 may have access to a real-time streamed or mirroredview of the VR or AR experience 110 via a second user device. The seconduser device may be configured to execute a mobile companion application105 that is programmed to display the real-time streamed or mirroredview of the VR or AR experience 110. The mobile companion application105 may comprise software programmed to execute the present disclosure.The second user device may be any device comprising one or moredisplays, such as described above. In some instances, the second userdevice may be any device comprising a user interactive device (e.g.,buttons, touchscreen, keyboard, etc.) that allows the supervisor 104 toprovide an input to the second user device. In some cases, the seconduser device may provide the supervisor 104 with a non-stereoscopicalview of the user's 102 experience 110. For example, the mobile companionapplication 105 may display the experience 110 in a wider field of viewthan is provided to the user 102 so that the supervisor 104 can see whatis to the right and left of the user's view. As an alternative, thesupervisor 104 may be provided with an extended view to that of the user102 in all angles including in the diagonal, up, and down directions. Asan alternative, the mobile companion application 105 of the supervisor104 may provide such content asynchronously, such as at a later time.The mobile companion application 105 may further provide a console 130which allows the supervisor 104 to intervene (e.g., trigger actions orprompts) in the user's 102 virtual reality or augmented reality scene125. Such interventions may include manipulating characters in thescene, triggering audio, visual, and/or haptic change in the VR or ARenvironment, or taking administrative action such as starting a newlearning module or ending the therapy session.

A control system 106 may provide content for the user's 102 virtualreality experience 110. In some instances, the content may be updated,modified, created, and/or removed, such as by the supervisor 104 and/oranother individual or entity. The control system 106 may be hardwareand/or software. In some instances, the control system 106 can reside onthe first user device, such as as an application or a program. In someinstances, the control system 106 can reside on the second user device,such as as an application or a program. In some instances, the controlsystem 106 may reside on a server and/or communicate via one or morecomputer networks with the first user device and/or the second userdevice. In some instances, the control system 106 may be distributedacross one or more devices (e.g., server hardware, first user device,second user device, etc.) which individually, or collectively, performthe operations of the control system 106. The control system 106 can bea computer control system, as described further below. In some cases,the control system 106 may pair the mobile companion application of thesupervisor 104 with the VR or AR experience 110 of the user 102. Forexample, the mirroring of the VR or AR experience 110, and/or the one ormore commands from the console 130 to the VR or AR experience 110 may becommunicated between the first user device and the second user devicethrough the control system 106.

Data collected about the user 102 with relation to the VR or experience110, such as from one or more sensors, may be stored and/or transmittedto the first user device, the second user device, and/or the controlsystem 106.

FIG. 2 shows an illustration of what a mobile companion application maydisplay to a supervisor of a user. The mobile companion application 200may comprise “field of view” markers 210 distinguishing the boundarybetween what the user is viewing and the extended view of thesupervisor. These “field of view” markers 210 need not be limited to theleft and right boundaries, and, in the alternative, can mark theboundaries in other directions, including the diagonal, up, or downdirections. The supervisor may further be provided with a selection oflearning cards 220, 225, 230 (e.g., “Intro,” “Explore,” “Look—Elephant,”“Giraffe”), including actions 235 to trigger (e.g., “Point”) andthumbnail pictures 240 of the selection of learning cards 220, 225, and230. The companion application display may also include an indication ofthe learning module 205 that the user is “playing” and a display ofprogress checking mechanisms 215 such as a safety timer.

In one embodiment, more devices may be paired, as needed, for increasedmonitoring. For example, a second companion application that provides acamera view can be used as an in-room monitor. A therapist may view theuser in a VR or AR session through a small video window on the companionapplication interface. For example, a user device of a user may bepaired with at least about 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30,35, 40, 45, 50 or more devices of supervisors, each having their owndevice. Alternatively or in addition, a user device of a user may bepaired with at most about 50, 45, 40, 35, 30, 25, 20, 15, 10, 9, 8, 7,6, 5, 4, 3, or 2 devices of supervisors. In some cases, a supervisor maybe paired with a plurality of users. For example, a user device of asupervisor may be paired with at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 15,20, 25, 30, 35, 40, 45, 50 or more devices of users, each having theirown device. Alternatively or in addition, a user device of a supervisormay be paired with at most about 50, 45, 40, 35, 30, 25, 20, 15, 10, 9,8, 7, 6, 5, 4, 3, or 2 devices of users. In some cases, a virtualreality experience may allow a plurality of users (e.g., peer subjects)to interact with each other in the virtual reality experience, such asfor a group therapy session.

In one embodiment, the pairing of these devices (e.g., devices forsupervisors, devices for users, etc.) may be remote which allows forteletherapy sessions. The devices can be paired over a network, such asover the internet, intranet, and/or extranet, so that the companionapplication may operate remotely from the user.

In one embodiment, the system can use a gaze tracking algorithm todetermine the object of focus for the user and trigger actions andprompts based on that gaze. The system may use such tracking algorithmto collect metrics and provide an analytics dashboard comprising datasuch as reaction time to a task, whether the user is taking fixationbrakes, gaze accuracy, and how steady their gaze is on an object versusa neurotypical user. Alternatively, the system may compare the user withother subjects receiving the same or similar treatments. These analyticsmay provide not only the metrics tracked during the user's session butalso a longitudinal view of the user's performance progress. See FIG. 7for an illustration of an analytics dashboard. This dashboard may beused as a user progress report and contain information such as taskcategory, task name, start date, total number of sessions, total numberof minutes, total number of trials, total number of successes, successrate, best week, user profile data including such things as, forexample, user age and gender, and a graphical illustration of theprogress of the user such as, for example, user's reaction time, gazeaccuracy, gaze stability, and fixation breaks. The dashboard may containsupplemental analytics on each progress category based on the goals ofthe training objective (e.g., Total Minutes: “Exceeds 10 minutes persession effective session duration”; Total Trials: “Meets minimumeffective repetitions per session”; Success Rate: “Exceeds expected 75%success rate”, Total Sessions: “18 left out of 150 Recommended Total”).The dashboard may contain other sensory data, such as measured by theone or more sensors described elsewhere herein. The sensory data can beindicative of a subject's progress toward one or more therapeutic goals,such as a reaction time, gaze stability, gaze accuracy, response volume,the presence or lack of a movement, the presence or lack of speaking, orother outputs by the subject. The sensory data can comprise dataotherwise undetectable or difficult to detect or quantify withtraditional therapy methods.

In one embodiment, the system can integrate motion controllers into thetherapy modules to track the user's motions in the virtual or augmentedreality system. For instance, in a joint attention module where the useris evaluated on his or her ability to respond to pointing, there may bea lesson where the user can point in real space holding a controller.The system can then track where the gesture lands in the virtual oraugmented reality space and base actions and prompts accordingly. Inanother instance, in a module where the user is trained for interactionwith the police, the same motion tracking capability may track the handposition of the user to determine if the user is exhibiting behaviorthat may put them in danger with the police.

In one embodiment, the system can integrate voice recognition technologyto enable therapeutic conversation training for the user in the VR or ARspace. For instance, a user may be given simple scenarios where the useranswers the door and finds themselves having a back and forthconversation with a virtual character that leverages voice recognitionmixed with override input from the supervisor to create a coherenttraining environment.

Virtual Reality Content

Translating therapeutic goals and abstract processes into concretevirtual reality or augmented reality may be a multidisciplinary endeavorwhich can involve the coordination and communication of experts in asmany fields as psychology, medicine, speech-language therapy,occupational therapy, physical therapy, behavioral intervention, gamedesign, software development, and visual and auditory arts. The systemcan facilitate such multidisciplinary coordination by systematicallycategorizing the therapeutic content into distinct units, such as“learning modules” and “learning cards.” A learning module can representa developmental milestone objective for the user. It can comprise adistinct set of therapeutic content that attempts to improve one or moretarget skills in the user. In some aspects, the system may comprise acollection of learning modules from which a supervisor may select to optinto to “play” with a user.

FIG. 3 depicts a logical model for a learning module 305. In FIG. 3,each line ending with a circle denotes a “one to many” relationship, inwhich a line connector represents “one” and a circle connectorrepresents “many.” For example, a line ending with circles on both endswill denote a “one to many” relationship in both directions. Eachtriangle denotes an “is a” relationship.

In this logical model, each game scene 335 may comprise one or morelearning modules 305, one or more models 340 (e.g., avatar, 2-D or 3-Danimated characters), and one or more audio 350. Each learning module305 may then comprise one or more learning cards 310. Each learning card310 may comprise one or more models 340 and one or more sequences 315.Sequences 315, which can be triggered by user events 320 or applicationprompts 325, can comprise one or more events 320 as well as one or moreprompts 325. Each of the events 320 and the prompts 325 may in turncomprise one or more sequences 315. That is, for example, a sequence 315may trigger an event 320, and the event 320 may also trigger a sequence315. Additionally, each of the event 320 and the prompt 325 may compriseone or more audio/visual updates 330 (“A/V updates”). Each A/V update330 may comprise a banner update 355 in the supervisor's application aswell as an animation 345 and audio 350 element in the game scene 335.One model 340 (e.g., an animal character) may relate to one or more gamescenes 335 (e.g., zoo, train station), and comprise one or moreanimation 345 elements and one or more audio 350 elements. An audio 350element may comprise one or more models 340.

As discussed briefly above, a learning module 305 can represent adevelopmental milestone in the form of a distinct set of therapeuticcontent that attempts to improve one or more target skills in the user.A supervisor may opt into a learning module 305 to “play” with a user.Each learning module 305 may be defined by the following parameters:title (e.g., “Safari”), educational focus (e.g., joint attention), unityscene (e.g., “Safari”), thumbnail image, and a collection of learningcards 310, including an Intro Learning Card. A unity scene, or the gamescene 335, may comprise the game board, including the scene backgroundand audio 350 elements independent of the models 340 (e.g., introductorymusic, generic background sounds), and a collection of two or threedimensional models 340, which normally includes the supervisor's avatar.

A learning card 310 can be a distinct playable unit within a learningmodule 305 that has a set of learning objectives. That is, a learningcard 310 may represent a single exercise that helps move the user towardthe milestone encompassed by the learning module 305. A learning card310 may define which of a game scene's 335 models 340 are present andinclude a collection of sequences 315. These sequences 315 may interactboth with a supervisor's companion application, such as through bannerupdates 355, and with the game scene 335 and models 340, such as throughthe animation 345 and audio 350 elements. Each learning card 310 may bedefined by the following parameters: title, thumbnail image, acollection of sequences 315, and educational description, which is along-form content that describes the purposes of the particular learningcard 310. The logical flow of a learning card 310 is discussed infurther detail with the discussion of FIG. 4.

A sequence 315 may be a collection of prompts 325 and events 320 thattogether progresses the user through some or all of the objectives ofthe learning card 310. A sequence 315 may be triggered either by anevent 320 (e.g., the user fixes their gaze on an animal) or by an actionprompt 325 on the learning card 310 which is initiated by a supervisor(e.g., the supervisor presses a button to “Point” at a Giraffe model340). A sequence 315 may be defined by the following parameters: theevent 320 or prompt 325 that triggers the sequence 315 and thecollection of events 320 and prompts 325 within the sequence 315.

An event 320 may be a user action that may be explicitly captured by theVR or AR system for the purposes of triggering A/V updates 330 and/orupdating active sequences 315 that contain the event 320. An event 320may be defined by the following parameters: the user's triggering action(e.g., maintaining gaze on the Pig model 340 for 2 seconds), and adefined set of A/V updates 330 that execute as a post-action to thetriggering action (e.g., a banner 355 status update in the supervisor'scompanion application reading, “User has gazed at Pig,” or the Pig model340 initiating an audio 350 element such as “Oink” in the game scene335).

A prompt 325 may be a supervisor-initiated component of a sequence 315.It may be defined by the following parameters: a named action whichappears as a button for the learning card 310 (e.g., a “Point” button),clear logic for a user to complete the prompt 325, a collection of A/Vupdates 330 should the user fail to follow the given prompt 325 whichmay include a banner 355 status update to the supervisor (e.g., “Tellthe user to look at the Pig if they are not picking up the visualpointing cue” displayed to the supervisor in a companion application) oran animation 345 or audio 350 update in the game scene 335 (e.g., Pigmodel 340 is animated 345 and/or makes a sound 350), a collection of A/Vupdates 330 should the user succeed in following the given prompt 325which may include a banner 355 status update to the supervisor (e.g.,“Congratulate user for looking at the Pig Model” displayed to thesupervisor in a companion application) or an animation 345 or audio 350update in the game scene 335 (e.g., Pig model 340 is animated 345 or anaudio 350 is initiated for celebratory music).

An A/V update 330 may be in the form of updates to the supervisorthrough a banner update 355, the triggering of animation 345 or audio350 of the models 340 in the game scene 335, or the triggering of audio350 in the game scene 335 level independent of the models 340. Forexample, an A/V update 330 may be defined by the following parameter:which of banner 355, animation 345, or audio 350 is to be updated (e.g.,in the case of a banner update 355, the text should be specified).

FIG. 4 shows a logical model for an illustrative learning card, element310 in FIG. 3, titled “Eye FollowU—LC 2: Alert to Sound Source” 402. The“Eye FollowU” learning card 402 is illustrative of the logicalcomponents and methods introduced in FIG. 3 and discloses but one designof many possible. “Eye FollowU” 402 is a Card whose educational focus isjoint attention. The skill within joint attention is alerting to a soundsource. The player profile for the Card 402, that is, the type ofaffected individual for which the Card 402 is intended, is an individualwho does not readily respond to the sound in his or her environment andlikely has difficulty responding when his or her name is called. Thebehavioral goal for the Card 402 is for the user to shift his or hergaze to look at an animal model, element 340 in FIG. 3, that hasproduced a sound and hold the gaze on that animal for at least 2seconds. The Card 402 may be completed when the user demonstrates thisbehavior at least 5 times. The Card may have the following parameters:Unity Scene (Safari); Models (Elephant, Gorilla, Adult Avatar);Thumbnail Image (figure with hand cupping ear); Sequence ([Animal][makes appropriate animal sound]).

In an example, the Card 402 may separately provide the following notesand instructions to the supervisor in the companion application:

-   -   “This Card is for teaching the player to alert to the sound of        the animal, a skill which he or she will need as he progresses        through the Learning Cards. This is not intended as a        ‘directions following’ game. So, don't tell the player to look        for an animal. Let the program do the work and then join in the        fun.”    -   “LISTEN for the animal sound. WAIT for the player to find the        animal. THEN, respond, offering clues only if necessary.”    -   “Feel free to give verbal praise or comments after the Player        looks at the animal.”    -   “If the player is having trouble identifying the animal that        goes with the sound, try going back to the exploratory level.”

FIG. 4 illustrates the possible logical flow paths of this Learning Card402 in action. In FIG. 4, each solid black arrow represents a definitesequence and each broken arrow represents one of many sequential optionsthat are available.

In the logical design of the Learning Card 402, producers of the Card,such as the developers or designers, may highlight a “Happy Path” 404.The Path 404 may highlight (e.g., bold, color) the expected sequentialpath of a successful user, that is, a user who successfully completesthe learning objective task given by the learning card. For the “EyeFollowU” Card 402, the execution sequence 406-422 is bolded as the HappyPath 404. Such highlighting may be made visible only within the system'sinternal design process, such as to developers or designers of the Card402, for reference and design purposes.

The Card's 402 first event 406 occurs when the user enters the VR or ARscene. Upon the occurrence of this event 406, the supervisor receives abanner update 408 on the companion application reading, “Press ‘Begin’when you are ready to start.” The supervisor initiates a prompt 410 whenthe supervisor presses the “Begin” button on his companion application.Upon initiation of the prompt 410, there is an A/V update 412 in the VRor AR scene, and an animal model that is off-screen makes a noise.Simultaneously, the A/V update 412 starts a timer 414 to collect metricson the user. After hearing the animal noise, the user may proceed withone of two events, the first event 416 in which the user successfullyshifts his or her gaze to the noise-making animal model, and the secondevent 424 in which the user fails to look at the animal model. Theuser's success 416 in shifting his or her gaze initiates an A/V update418 in which the supervisor receives a banner update 420 reading, “Offercomment or praise: ‘Wow!’; ‘Nice looking,’” and in the game scene theanimal model goes into 1 to 2 seconds of “Ready” mode, followed by areward animation 418. An example of an animal model in “Ready” mode 418may be a three-dimensional elephant model jumping on all fours for 2seconds. After the reward animation, the Card is marked completed “[x]of 10 times,” 422 and the same message is displayed in the supervisor'scompanion application in a banner update. Alternatively, in otherdesigns, the Card can be marked completed only after a user hascompleted the task a [y] number of times. If, on the other hand, theuser fails to look at the animal model that made the noise 424, anotherA/V update 426 is initiated where the animal model begins to move closerto the user's field of view, but still remains out of sight. This update426 triggers another program event 428 which starts a new timer thatmeasures the duration of time from the beginning of the user's walk toeither the time the user looks at the animal model or the time theanimal model makes the next sound. After the animal model moves closer,in another A/V update 430, the sound reinitiates, this time longer induration and possibly louder in volume. At this point, again, the usercan proceed with one of two events, the first event 416 in which theuser successfully shifts gaze to the noise-making animal model, and thesecond event 432 in which the user fails for a second time. The user'ssuccess will trigger the same sequence of A/V updates and events 416-422as when the user succeeded on gazing at the first instance of the noise.If the user fails, another A/V update 434 is triggered, and the animalmodel is made to move even closer to the user's field of view, but nowhalf or partly in the field of view of the user. After the animal modelhas moved partly within the view of the user, in another A/V update 438,the sound reinitiates, in longer duration and louder volume than thesecond instance of the noise. Another timer starts 436 measuring theduration of time from the beginning of the user's walk until either theuser shifts his or her gaze on the animal model or the animal modelmakes the next sound. At this point, again, the user may proceed withone of two events, the first event 416 in which the user successfullyshifts gaze to the noise-making animal model, and the second event 440in which the user fails for the third time. The user's success willtrigger the same sequence of A/V updates and events 416-422 as when theuser succeeded on gazing at the first instance of the noise. This time,however, the user's failure will trigger a banner update 442 in thesupervisor's companion application that reads “Consider going back tothe previous level.” From this logical model, the system can collect anumber of data points including the number of times a user alerts to thesound source, which animal models the user responds to, the time ittakes the user to alert to the sound source, and a ratio between thenumber of times the user had the opportunity to react to the sound tothe number of times the user actually reacted. Such data may bepresented to a supervisor as a progress report of the dashboard formatillustrated in FIG. 7 or, alternatively, as any other format or report.

In other designs, the number of iterations, or attempts given to theuser, for a certain learning exercise may be modified as needed anddepending on the difficulty level of the particular learning card orlearning module.

In some aspects, the system may provide teaching modules which teachfunctional skills using stories. These teaching modules can includeplacing a subject in a practice environment for both routine andnon-routine tasks. Beneficially, such stories provide an effective andentertaining solution to teach subjects how to navigate everydayinteractions by placing the subjects in a controlled virtual oraugmented reality environment, thereby shielding the subjects frompotential actual harms and retaining control to effectively guide thesubjects at a flexible pace.

FIG. 5 shows an illustration of a teaching module display for asupervisor. The teaching module can be a learning module, as describedelsewhere herein. The teaching module can comprise a practiceenvironment for interacting with a police officer that a user meets inan urban street. For example, the teaching module can be configured toteach users conversation skills by prompting them to answer questionsfrom a police officer avatar that approaches them on an urban street. Acompanion application of a supervisor may comprise an interface 500 tocontrol the avatar. The interface 500 may include a mirrored view 505 ofthe user's VR or AR experience. The mirrored view 505 of the user's VRor AR experience may be a non-stereoscopical view and/or an extendedview of the user's VR or AR experience. The interface 500 may compriseone or more guidelines to distinguish the user's view and the extendedview. The interface 500 may comprise one or more prompts 515 displayedto the user.

The interface 500 may comprise a console 510 allowing the supervisor totrigger or prompt the user, such as by controlling the police officeravatar. For example, the supervisor may select an option to speak upwhen the response is not clear (e.g., control avatar to say “Can you saythat again?”), select an option to require more information when theresponse is incomplete (e.g., control avatar to say “I need moreinformation”), or select an option to repeat a question or ask aconfirmatory question when the subject is not responsive or gives anirrelevant response (e.g., control avatar to say “Hey there!” or “Didyou hear me?”).

In some instances, the interface 500 may allow the supervisor to recorda question (or other conversation piece) in the supervisor's voice viathe supervisor's user device and transmit and playback the recordedquestion (or other conversation piece) to the user via the user's userdevice (e.g., as the police office avatar). In some instances, theinterface 500 may allow the supervisor to input text, and use one ormore text-to-speech converters to read the text as the police officeavatar to the user. In some instances, the interface 500 may perform aspeech-to-text conversion and then a text-to-speech conversion toconvert the voice/tone of the supervisor in a recording. In someinstances, the teaching module may be programmed, such as to execute oneor more algorithms, to recognize a certain type of response from thesubject and provide an automatic response without a command from thesupervisor. For example, if an audio sensor (e.g., microphone) collectsdata from the subject's response and determines that the volume is toosmall, the teaching module can be programmed to have the police officerrespond with “Can you say that again?” without user input from thesupervisor. In such case, a supervisor can override pre-programmedresponses. In another example, if an audio sensor collects data that thesubject has failed to respond to a question by the police office avatarfor a certain duration (e.g., 30 seconds, 1 minute, etc.), the teachingmodule can be programmed to have the police officer respond with “Didyou hear me?” In another example, if an audio sensor collects data fromthe subject's response, the response can be processed via aspeech-to-text conversion and compared to a model text answer. If thecomparison yields a % similarity above a predetermined threshold, theteaching module can record the conversation session as a successfulconversation.

In some aspects, provided are sensory-based therapies. Sensory-basedtherapies can include using a VR or AR environment to build specificcalming or stimulating experiences with a focus on helping users withneurodevelopmental disorders or disabilities, such as autism, to manageor respond to otherwise challenging environments and situations. Forexample, a calming module may allow a user to play a simple anddelightful musical instrument using just the direction of his or hergaze. In some instances, one or more VR or AR experiences may bedesigned for specific calming requirements. Such VR or AR experiencesmay comprise environments such as an aquarium, a beach, a butterflypark, or a play room.

FIGS. 6A-6C illustrate a sensory module content comprising a train room.FIG. 6A shows a user 605 using a user device 610. FIG. 6B shows the user605 using the user device 610 immersed in a virtual reality or augmentedreality experience 615. FIG. 6C shows an exemplary display 620 of thesensory module as seen by the user. The sensory module comprising thetrain room may comprise one or more dynamic components capable ofinteracting with the user 605. For example, the user 605 may be capableof adding one or more train tracks using just the user's 605 gaze,receive colorful train beads as rewards, such as from a rewardsdispenser, for successfully completing one or more tasks in the module(e.g., adding a train track with the user's 605 gaze), selecting a typeof train track with the user's 605 gaze or other motion or output (e.g.,sound, voice command, etc.), and/or selecting a train model with theuser's 605 gaze or other motion or output (e.g., sound, voice, command,etc.). The train may be in animated to be in motion around the traintracks that the user 605 has built. The VR or AR experience 615 may be a360° experience, wherein the user may view content in 360°.

FIG. 6D illustrates a sensory module content comprising an aquarium. Auser 625 using a user device 620 is immersed in a virtual reality oraugmented reality experience 630. The sensory module comprising theaquarium may comprise one or more dynamic components capable ofinteracting with the user 625. For example, the user 625 may be capableof adding one or more fish inside the aquarium using just the user's 605gaze. The user 625 may be capable of choosing different environments.The VR or AR experience 630 may be a 360° experience, wherein the usermay view content in 360°.

FIG. 6E illustrates a sensory module content comprising a butterflypark. A user 645 using a user device 635 is immersed in a virtualreality or augmented reality experience 640. The sensory modulecomprising the aquarium may comprise one or more dynamic componentscapable of interacting with the user 645. For example, the user 645 maybe capable of adding one or more animated butterflies inside thebutterfly park using just the user's 645 gaze, and/or playing a music inthe background using gaze. The user 645 may be capable of choosingdifferent environments. The VR or AR experience 640 may be a 360°experience, wherein the user may view content in 360°.

FIG. 6F illustrates a sensory module content comprising a lava room. Auser 660 using a user device 655 is immersed in a virtual reality oraugmented reality experience 650. The sensory module comprising the lavaroom may comprise one or more dynamic components capable of interactingwith the user 660. For example, the user 660 may be capable ofcontrolling random colored balls with just the gaze. The user 660 may becapable of choosing different environments. The VR or AR experience 650may be a 360° experience, wherein the user may view content in 360°.

FIG. 6G illustrates a sensory module content comprising a coloring book.A user 675 using a user device 665 is immersed in a virtual reality oraugmented reality experience 670. The sensory module comprising thecoloring book may comprise one or more dynamic components capable ofinteracting with the user 675. For example, the user 675 may be capableof selecting different coloring books, select different coloringoutlines, color an outline with just the gaze, and/or select one or moretypes of pens or other tools, and parameters for the tools (e.g., width,transparency, color, etc.). The VR or AR experience 670 may be a 360°experience, wherein the user may view content in 360°.

In some aspects, provided are systems and methods for diagnosing a userwith a condition. In some cases, data collected (or recorded) for one ormore subjects may be aggregated to build behavior models for one or moreconditions (e.g., mental or developmental disorders). Such behaviormodels can be leveraged as diagnostic tools for users to be evaluatedthrough the VR or AR system. For example, a plurality of behavior modelsfor different mental or developmental disorders can be stored in alibrary of behavior models, such as in one or more databases. Aplurality of behavior models for each VR or AR environment, scene, orexperience may be stored in a library of behavior models, such as in oneor more databases. A behavior model for a first type of developmentaldisorder can comprise data exhibited by one or more subjects known tosuffer from the first type of developmental disorder when placed in afirst VR or AR environment. When a user to be diagnosed is placed in thesame first VR or AR environment, or an environment similar to the firstVR or AR environment, the data collected on the user may be compared tothe behavior model for the first type of developmental disorder todetermine whether the user has the first type of developmental disorderor not, and/or determine a degree to which the user suffers from thefirst type of developmental disorder. In some instances, the datacollected for the user to be diagnosed may be compared to a plurality ofbehavior models to determine which one or more conditions the user maysuffer from (and to what degree). By way of example, the higher the %similarity between the collected data for the user and the data storedfor the behavior model, the more likely it is (and with higher degree)that the user suffers from the condition of the behavior model. In someinstances, a user being diagnosed may be placed in a plurality ofdifferent VR or AR environments, scenes, and/or experiences (e.g.,sensory calming modules, teaching modules, learning modules, etc.) andan overall performance by the user may be compared to the library ofbehavior models. Such comparisons may be made individually by VR or ARenvironments (for each type of condition) and/or by condition (for eachtype of VR or AR environments), and then aggregated (e.g., average,mean, median, other statistical computation or evaluation).Beneficially, the VR or AR systems may accurately and precisely diagnosea user with a condition based on comparisons of accurately measuredempirical data (for example, compared to general guidelines or hunchesthat a child is not attentive enough), and determine a degree ofintensity or progression of a condition. This may be particularlybeneficial for diagnosing and treating mental or developmental disorderswhere it is difficult to quantify symptoms. In some instances, thediagnosis of the VR or AR systems may be implemented by one or morecomputer algorithms, such as machine learning algorithms, which aretrained with increasing data input (e.g., more subjects, more therapysessions using the VR or AR systems, etc.). For example, the accuracy ofthe diagnosis may increase as the iterations increases.

In some aspects, different virtual or augmented reality experiences maybe organized in a library and be available for prescription orrecommendation to care givers, parents, or the subjects themselves. Forexample, they may be prescribed or recommended by human expertsdiagnosing the subjects. Alternatively or in addition, they may beprescribed by the system via one or more computer algorithms (e.g.,machine learning algorithms), such as described above. In some cases,the diagnosis, prescription, or recommendation may be made without humaninput. The accuracy of the diagnosis, prescription, or recommendationmay increase with increasing iterations. In some instances, the virtualor augmented reality experiences and therapies can be tailored to theneeds of individual subjects either manually by the human expert (e.g.,therapist) or using computer algorithms. For example, the tailoring canbe performed based at least on prior conditions of the subject and/orbased on data collected throughout the subject's and others' use of theVR or AR system, such as described above.

The VR or AR system described herein may provide a low cost andaccessible therapeutic solution for subjects who have a mental ordevelopmental disorder. Alternatively or in addition, the VR or ARsystem described herein may provide educational and/or entertainmentvalue to subjects who have a mental or developmental disorder, andsubjects who do not.

Control Systems

The present disclosure provides computer control systems that areprogrammed to implement methods of the disclosure. FIG. 8 shows acomputer system 801 that is programmed or otherwise configured tooperate and display a virtual reality or augmented reality environmentwith or without the use of supplemental headsets or gear, executealgorithms within the virtual reality or augmented reality platform or acompanion application to the virtual reality or augmented realityplatform, track, save, and analyze a user's behaviors and responses in aVR or AR environment, and execute and pair a companion application thatmay allow a supervising user to monitor and control in real-time auser's virtual reality or augmented reality experience on a second,third, or n^(th) display screen. The computer system 801 can be anelectronic device of a user or a computer system that is remotelylocated with respect to the electronic device. The electronic device canbe a mobile electronic device. The computer system 801 may be used totreat a neurodegenerative disorder, such as, for example, autism. Thecomputer system 601 may communicate with the control system 106 ofFIG. 1. In some cases, the control system 106 may comprise the computersystem 801.

The computer system 801 includes a central processing unit (CPU, also“processor” and “computer processor” herein) 805, which can be a singlecore or multi core processor, or a plurality of processors for parallelprocessing. The computer system 801 also includes memory or memorylocation 810 (e.g., random-access memory, read-only memory, flashmemory), electronic storage unit 815 (e.g., hard disk), communicationinterface 820 (e.g., network adapter) for communicating with one or moreother systems, and peripheral devices 825, such as cache, other memory,data storage and/or electronic display adapters. The memory 810, storageunit 815, interface 820 and peripheral devices 825 are in communicationwith the CPU 805 through a communication bus (solid lines), such as amotherboard. The storage unit 815 can be a data storage unit (or datarepository) for storing data. The computer system 801 can be operativelycoupled to a computer network (“network”) 830 with the aid of thecommunication interface 820. The network 830 can be the Internet, aninternet and/or extranet, or an intranet and/or extranet that is incommunication with the Internet. The network 830 in some cases is atelecommunication and/or data network. The network 830 can include oneor more computer servers, which can enable distributed computing, suchas cloud computing. The network 830, in some cases with the aid of thecomputer system 801, can implement a peer-to-peer network, which mayenable devices coupled to the computer system 801 to behave as a clientor a server.

The CPU 805 can execute a sequence of machine-readable instructions,which can be embodied in a program or software. The instructions may bestored in a memory location, such as the memory 810. The instructionscan be directed to the CPU 805, which can subsequently program orotherwise configure the CPU 805 to implement methods of the presentdisclosure. Examples of operations performed by the CPU 805 can includefetch, decode, execute, and writeback.

The CPU 805 can be part of a circuit, such as an integrated circuit. Oneor more other components of the system 801 can be included in thecircuit. In some cases, the circuit is an application specificintegrated circuit (ASIC).

The storage unit 815 can store files, such as drivers, libraries andsaved programs. The storage unit 815 can store user data, e.g., userpreferences and user programs. The computer system 801 in some cases caninclude one or more additional data storage units that are external tothe computer system 801, such as located on a remote server that is incommunication with the computer system 801 through an intranet or theInternet.

The computer system 801 can communicate with one or more remote computersystems through the network 830. For instance, the computer system 801can communicate with a remote computer system of a user (e.g., VR or ARuser, supervisor, therapist). Examples of remote computer systemsinclude personal computers (e.g., portable PC), slate or tablet PC's(e.g., Apple® iPad, Samsung® Galaxy Tab), telephones, Smart phones(e.g., Apple® iPhone, Android-enabled device, Blackberry®), or personaldigital assistants. The user can access the computer system 801 via thenetwork 830.

Methods as described herein can be implemented by way of machine (e.g.,computer processor) executable code stored on an electronic storagelocation of the computer system 801, such as, for example, on the memory810 or electronic storage unit 815. The machine executable or machinereadable code can be provided in the form of software. During use, thecode can be executed by the processor 805. In some cases, the code canbe retrieved from the storage unit 815 and stored on the memory 810 forready access by the processor 805. In some situations, the electronicstorage unit 815 can be precluded, and machine-executable instructionsare stored on memory 810.

The code can be pre-compiled and configured for use with a machinehaving a processor adapted to execute the code, or can be compiledduring runtime. The code can be supplied in a programming language thatcan be selected to enable the code to execute in a pre-compiled oras-compiled fashion.

Aspects of the systems and methods provided herein, such as the computersystem 801, can be embodied in programming. Various aspects of thetechnology may be thought of as “products” or “articles of manufacture”typically in the form of machine (or processor) executable code and/orassociated data that is carried on or embodied in a type of machinereadable medium. Machine-executable code can be stored on an electronicstorage unit, such as memory (e.g., read-only memory, random-accessmemory, flash memory) or a hard disk. “Storage” type media can includeany or all of the tangible memory of the computers, processors or thelike, or associated modules thereof, such as various semiconductormemories, tape drives, disk drives and the like, which may providenon-transitory storage at any time for the software programming. All orportions of the software may at times be communicated through theInternet or various other telecommunication networks. Suchcommunications, for example, may enable loading of the software from onecomputer or processor into another, for example, from a managementserver or host computer into the computer platform of an applicationserver. Thus, another type of media that may bear the software elementsincludes optical, electrical and electromagnetic waves, such as usedacross physical interfaces between local devices, through wired andoptical landline networks and over various air-links. The physicalelements that carry such waves, such as wired or wireless links, opticallinks or the like, also may be considered as media bearing the software.As used herein, unless restricted to non-transitory, tangible “storage”media, terms such as computer or machine “readable medium” refer to anymedium that participates in providing instructions to a processor forexecution.

Hence, a machine readable medium, such as computer-executable code, maytake many forms, including but not limited to, a tangible storagemedium, a carrier wave medium or physical transmission medium.Non-volatile storage media include, for example, optical or magneticdisks, such as any of the storage devices in any computer(s) or thelike, such as may be used to implement the databases, etc. shown in thedrawings. Volatile storage media include dynamic memory, such as mainmemory of such a computer platform. Tangible transmission media includecoaxial cables; copper wire and fiber optics, including the wires thatcomprise a bus within a computer system. Carrier-wave transmission mediamay take the form of electric or electromagnetic signals, or acoustic orlight waves such as those generated during radio frequency (RF) andinfrared (IR) data communications. Common forms of computer-readablemedia therefore include for example: a floppy disk, a flexible disk,hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD orDVD-ROM, any other optical medium, punch cards paper tape, any otherphysical storage medium with patterns of holes, a RAM, a ROM, a PROM andEPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wavetransporting data or instructions, cables or links transporting such acarrier wave, or any other medium from which a computer may readprogramming code and/or data. Many of these forms of computer readablemedia may be involved in carrying one or more sequences of one or moreinstructions to a processor for execution.

The computer system 801 can include or be in communication with anelectronic display 835 that comprises a user interface (UI) 840 forproviding, for example, images or videos used to simulate an augmentedor virtual reality experience to the user, a mobile companionapplication to the supervisor, or an in-room monitor to therapists.Examples of UI's include, without limitation, a graphical user interface(GUI) and web-based user interface.

Methods and systems of the present disclosure can be implemented by wayof one or more algorithms. An algorithm can be implemented by way ofsoftware upon execution by the central processing unit 805. Thealgorithm can, for example, track a user's activity, including voice,motion, and gaze within the virtual reality or augmented realityenvironment, run analytics on metrics collected from a user'sexperience, provide therapeutic learning by iterating trainingprinciples with varying levels of difficulty, and translate commandsfrom a paired control device.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. It is not intendedthat the invention be limited by the specific examples provided withinthe specification. While the invention has been described with referenceto the aforementioned specification, the descriptions and illustrationsof the embodiments herein are not meant to be construed in a limitingsense. Numerous variations, changes, and substitutions will now occur tothose skilled in the art without departing from the invention.Furthermore, it shall be understood that all aspects of the inventionare not limited to the specific depictions, configurations or relativeproportions set forth herein which depend upon a variety of conditionsand variables. It should be understood that various alternatives to theembodiments of the invention described herein may be employed inpracticing the invention. It is therefore contemplated that theinvention shall also cover any such alternatives, modifications,variations or equivalents. It is intended that the following claimsdefine the scope of the invention and that methods and structures withinthe scope of these claims and their equivalents be covered thereby.

What is claimed is:
 1. A method for diagnosing a user with a mental ordevelopmental condition, comprising: (a) providing a first user having afirst mental or developmental condition a virtual or augmentedexperience, wherein the virtual or augmented experience provides one ormore of a visual, auditory, and haptic stimulation to the first user;(b) generating a first set of sensory data of the first user in responseto the one or more of the visual, auditory, and haptic stimulation; (c)associating the first set of sensory data to the first mental ordevelopmental condition; (d) providing a second user the virtual oraugmented experience; (e) generating a second set of sensory data of thesecond user in response to the one or more of the visual, auditory, andhaptic stimulation; and (f) processing the second set of sensory data tothe first set of sensory data to diagnose the second user with the firstmental or developmental condition or lack thereof.
 2. The method ofclaim 1, wherein (f) comprises diagnosing the second user with a degreeto which the second user suffers from the first mental or developmentalcondition.
 3. The method of claim 1, wherein (f) comprises determining a% similarity between the first set of sensory data and the second set ofsensory data.
 4. The method of claim 1, further comprising: in (a),providing a plurality of users each having the first mental ordevelopmental condition, including the first user and not including thesecond user, the virtual or augmented experience; in (b), generating anaggregate data set of the plurality of users in response to the one ormore of the visual, auditory, and haptic stimulation in the virtual oraugmented experience, wherein the aggregate data set comprises, or isbased at least in part on, said first set of sensory data; in (c),associating the aggregate data set to the first mental or developmentalcondition; and in (f), processing the second set of sensory data to theaggregate data set to diagnose the second user with the first mental ordevelopmental condition or lack thereof.
 5. The method of claim 1,further comprising: (g) providing a third user having a second mental ordevelopmental condition the virtual or augmented experience, wherein thesecond mental or developmental condition is different from the firstmental or developmental condition; (h) generating a third set of sensorydata of the third user in response to the one or more of the visual,auditory, and haptic stimulation; (i) associating the third set ofsensory data to the third mental or developmental condition; and (j)processing the second set of sensory data to the third set of sensorydata to diagnose the second user with the second mental or developmentalcondition or lack thereof.
 6. The method of claim 1, wherein the virtualor augmented experience permits the first user to interact with atherapist, parent, or peer.
 7. The method of claim 6, wherein thetherapist, parent, or peer is located remote from the first user.
 8. Themethod of claim 6, wherein the one or more of the visual, auditory, andhaptic stimulation is provided or instructed by the therapist, parent orpeer.
 9. The method of claim 1, further comprising: in (a), providingthe first user with a second virtual or augmented experience, whereinthe second virtual or augmented experience provides one or more of asecond visual, auditory, and haptic stimulation to the first user; in(b), generating an aggregate data set from (i) the first set of sensorydata of the first user in response to the one or more of visual,auditory, and haptic stimulation in the virtual or augmented experience,and (ii) a third set of sensory data of the first user in response tothe one or more of the second visual, auditory, and haptic stimulationin the second virtual or augmented experience; in (c), associating theaggregate data set to the first mental or developmental condition; in(d), further providing the second user the second virtual or augmentedexperience; in (e), generating a second aggregate data set from (i) thesecond set of sensory data of the second user in response to the one ormore of visual, auditory, and haptic stimulation in the virtual oraugmented experience, and (ii) a fourth set of sensory data of thesecond user in response to the one or more of the second visual,auditory, and haptic stimulation in the second virtual or augmentedexperience in (f), processing the second aggregate data set to theaggregate data set to diagnose the second user with the first mental ordevelopmental condition or lack thereof.
 10. A method for diagnosing auser with a mental or developmental condition, comprising: (a) providinga first user with one or more virtual or augmented experiences, whereinthe one or more virtual or augmented experiences is configured toprovide the first user one or more of a visual, auditory, and hapticstimulation to the first user; (b) generating a first set of sensorydata of the first user in response to the one or more of the visual,auditory, and haptic stimulation in the one or more virtual or augmentedexperiences; and (c) processing the first set of sensory data of thefirst user to one or more behavior models in a library of behaviormodels, wherein a behavior model of the one or more behavior models isassociated with (i) a mental or developmental condition and (ii) the oneor more virtual or augmented experiences, to diagnose the first userwith the mental or developmental condition or lack thereof.
 11. Themethod of claim 10, wherein the behavior model is generated from sensorydata collected from a second user having the mental or developmentalcondition in response to the one or more of the visual, auditory, andhaptic stimulation provided to the second user in the one or morevirtual or augmented experiences.
 12. The method of claim 11, whereinthe behavior model is generated from sensory data collected from aplurality of users, including the second user and not including thefirst user, each having the mental or developmental condition, inresponse to the one or more of the visual, auditory, and hapticstimulation provided to the plurality of users in the one or morevirtual or augmented experiences.
 13. The method of claim 10, wherein(c) comprises diagnosing the first user with a degree to which the firstuser suffers from the mental or developmental condition.
 14. The methodof claim 10, wherein (c) comprises determining a % similarity betweenthe first set of sensory data and a set of sensory data associated withthe behavior model.
 15. The method of claim 10, wherein the one or morevirtual or augmented experiences permit the first user to interact witha therapist, parent, or peer.
 16. The method of claim 15, wherein thetherapist, parent, or peer is located remote from the first user. 17.The method of claim 15, wherein the one or more of the visual, auditory,and haptic stimulation is provided or instructed by the therapist,parent or peer.
 18. The method of claim 10, wherein each behavior modelin the library of behavior models is associated with (i) a given mentalor developmental condition and (ii) a given one or more virtual oraugmented experiences.
 19. The method of claim 10, wherein the behaviormodel is further associated with (i) a second mental or developmentalcondition different from the mental or developmental condition, (ii) anadditional virtual or augmented experience in addition to the one ormore virtual or augmented experiences, or both.
 20. The method of claim10, wherein (c) comprises processing the first set of sensory data ofthe first user to the one or more behavior models in the library ofbehavior models to diagnose the first user with a plurality of differenttypes of mental or developmental conditions, including the mental ordevelopmental condition, or lack thereof.